Demonstrate an understanding of literature exploration and ethical considerations in conducting research.
Lecture 1 established why a literature study matters. This lecture supplies the four methods by which one is actually conducted, and the rules for choosing between them.
Section 6 (the comparative table) is the single highest-yield artefact in this lecture: examiners test the quality appraisal and reproducibility rows most often, because that is where the four methods differ most consequentially.
The meta-analysis mathematics in Section 5 is examined conceptually. You must be able to say what each quantity measures and when each model is appropriate; numerical computation appears only in the practice question at the end.
1. Learning Objectives
- Define and distinguish the four principal approaches to literature review: systematic, narrative, scoping and meta-analytical.
- Identify the methodological steps required to conduct each type of review.
- Evaluate the appropriateness of a review approach for a given research question in computer science.
- Apply established reporting standards (PRISMA) to structure a literature review.
- Critically appraise published reviews for methodological rigour.
2. Introduction: The Review as a Method
A literature review is far more than a descriptive summary of prior work. It is a systematic, replicable and transparent synthesis of existing scholarly contributions within a defined domain. In computer science, where research output doubles approximately every five years, conducting a rigorous review is a foundational research competency.
The choice of approach shapes four things:
- The breadth and depth of evidence included.
- The reproducibility of the search and synthesis process.
- The type of conclusion that can legitimately be drawn — qualitative or quantitative.
- The defensibility of the resulting claims under peer review.
3. Foundational Concepts
3.1 The Review Protocol
A review protocol is a pre-specified plan that outlines the research question, inclusion and exclusion criteria, search strategy, data extraction methods and synthesis approach. Protocols enhance transparency and minimise bias.
Pre-registration of protocols — on PROSPERO in health research, or in an institutional archive or the Open Science Framework in computing — is becoming an expectation in empirical software engineering.
3.2 Structured Question Formats
| Framework | Components | Typical use |
|---|---|---|
| PICO | Population, Intervention, Comparison, Outcome | Systematic reviews with quantitative outcomes |
| PEO | Population, Exposure, Outcome | Scoping and narrative reviews of qualitative phenomena |
| PCC | Population/Participants, Concept, Context | Scoping reviews |
Population: undergraduate programming students. Intervention: automated feedback tooling. Comparison: manual teaching-assistant feedback. Outcome: assignment defect rate.
3.3 Reporting Standards
- PRISMA — Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
- PRISMA-S — the extension for reporting the search.
- PRISMA-ScR — the extension for scoping reviews.
A protocol governs what you will do; a reporting standard governs what you must disclose. Compliance with PRISMA does not make a weak review strong — it makes a weak review visible. Examiners test this distinction.
4. The Four Approaches
4.1 Systematic Literature Review (SLR)
A review that employs explicit, reproducible methods to collate and synthesise all studies that meet pre-defined eligibility criteria in response to a specific research question.
Methodological characteristics:
- Pre-registered protocol, often published or institutionally archived before screening begins.
- Exhaustive and reproducible search across multiple databases — IEEE Xplore, ACM Digital Library, Scopus, Web of Science, Google Scholar — supplemented by backward and forward citation chasing.
- Explicit inclusion and exclusion criteria applied through a two-phase screening process.
- Quality appraisal of included studies using validated tools (JBI Critical Appraisal Tools, CASP).
- Structured data extraction using standardised forms.
- Transparent reporting using PRISMA flow diagrams.
The PRISMA flow diagram documents records identified, screened, eligible and included, in four stages: Identification → Screening → Eligibility → Included. Writing Ni for records identified and Ne for records included:
where Nd,s, Nd,e and Nd,r denote records removed during screening, eligibility and final inclusion. This is a bookkeeping identity: it is examinable as a statement of completeness rather than as a computation.
The systematic review by Kitchenham et al. on evidence-based software engineering applied rigorous search strategies across seven digital libraries and produced the methodological guidelines that have become standard for software engineering SLRs.
| Strengths | Limitations |
|---|---|
| Reproducible and transparent | Time- and resource-intensive (6–24 months) |
| Reduces selection bias | May exclude contextually relevant grey literature |
| Enables evidence-based conclusions | Quality appraisal in CS is contested: designs are methodologically heterogeneous |
4.2 Narrative Literature Review
A narrative review — also called a traditional or qualitative review — provides a comprehensive, discursive synthesis of the literature without rigidly defined search protocols. It is interpretive rather than aggregative.
- Selection of sources rests on the author's expertise and judgement.
- No formal protocol is typically registered.
- Synthesis follows a thematic or chronological organisation.
- Quality appraisal is informal or absent.
- The aim is breadth, contextualisation and theoretical framing.
Three recognised variants: the editorial review (short, opinion-driven), the critical review (evaluative synthesis with an explicit stance), and the state-of-the-art review (current developments and future directions).
In computing, narrative reviews appear as survey articles in ACM Computing Surveys, as tutorial articles in IEEE Transactions introducing an emerging sub-field, and as the framing chapter of a thesis.
For a three-year-old subfield with forty heterogeneous papers and no comparable metrics, expert interpretive framing is the methodologically correct choice. The honest move is disclosure: state that selection was purposive rather than implying an exhaustiveness never attempted.
4.3 Scoping Review
A scoping review maps the key concepts, types of evidence and gaps in a research area, particularly where the topic has not been extensively reviewed or is complex and heterogeneous. Formalised by Arksey and O'Malley (2005) and refined by the Joanna Briggs Institute.
The Arksey and O'Malley framework specifies five stages, with an optional sixth:
- Identifying the research question — broad and often iterative.
- Identifying relevant studies — comprehensive but not exhaustive.
- Study selection — using post-hoc inclusion criteria.
- Charting the data — extracting key information in tabular form.
- Collating, summarising and reporting — descriptive and thematic.
- Consultation with stakeholders (optional) — adds interpretive value; in computing, practitioners routinely identify grey literature and deployment concerns absent from indexed venues.
Question formulation uses the PCC mnemonic — Population/Participants, Concept, Context — and reporting follows PRISMA-ScR. Scoping reviews suit mapping AI applications in healthcare, cataloguing privacy-preserving techniques in IoT, or charting trends in quantum computing.
A scoping review titled Machine Learning in Cybersecurity mapped ML applications across intrusion detection, malware classification and threat intelligence, identifying adversarial robustness in real-world deployments as an underexplored subdomain.
| Strengths | Limitations |
|---|---|
| Useful for broad, exploratory questions | Does not assess the methodological quality of included studies |
| Maps heterogeneity in a field | Findings may be less definitive |
| Identifies gaps for future SLRs | May overlap conceptually with narrative reviews |
4.4 Meta-Analysis
A meta-analysis is a quantitative synthesis that statistically combines the results of multiple independent studies addressing the same question, producing a pooled effect estimate. A systematic review is its precondition, not its alternative.
5. The Statistics of Pooling
5.1 Effect Sizes
For continuous outcomes the standardised mean difference is commonly used:
For binary outcomes the odds ratio is preferred, computed from the cell counts a, b, c, d of a 2×2 contingency table:
Standardisation is the conceptual key: dividing by the pooled standard deviation converts a raw millisecond or defect-count difference into a unitless quantity that can be averaged across studies using different scales.
5.2 Fixed-Effect and Random-Effects Models
The fixed-effect model assumes a single true effect size across studies; all divergence is sampling error:
The random-effects model assumes effect sizes vary across studies, incorporating between-study variance τ²:
Between-study variance is often estimated by the DerSimonian and Laird method:
Fixed effect asks what is the effect? Random effects asks what is the average of a distribution of effects? Because CS studies differ in datasets, hardware, hyperparameters and participant pools, random effects is the appropriate default.
Note that the models are nested: when τ² = 0 the random-effects weights collapse exactly onto the fixed-effect weights. Choosing fixed effect in the presence of real heterogeneity produces confidence intervals that are too narrow, and therefore overconfident conclusions.
5.3 Heterogeneity
Cochran's Q measures the weighted dispersion of study estimates about the fixed-effect pooled value, referred to a chi-square distribution on k − 1 degrees of freedom:
| I² value | Interpretation | What it obliges you to do |
|---|---|---|
| Below 25% | Low heterogeneity | Pooled estimate is readily interpretable |
| 25% to 75% | Moderate heterogeneity | Investigate sources; consider subgroups |
| 75% or above | High heterogeneity | Subgroup analysis or meta-regression — not a louder pooled claim |
Read I² as a ratio, not an absolute quantity: a large value with tightly clustered effects can be practically unimportant, while a modest value across wildly different effect magnitudes still matters. Always read it beside the forest plot.
5.4 Publication Bias
Publication bias is the tendency for studies with statistically significant results to be published preferentially. It is diagnosed by funnel plot asymmetry and tested formally by Egger's regression:
where b is the regression coefficient of the standard normal deviate on precision. In a symmetric funnel, precise studies cluster near the top around the pooled estimate and imprecise ones fan out below; a missing lower-left corner indicates small negative studies that were run but never published.
Benchmark culture rewards reporting improvements over null results, so negative findings on a popular architecture are systematically under-published. This links directly to the research-integrity material in Lectures 6 and 7: selective reporting is an ethics failure, not merely a statistical inconvenience.
6. Comparative Synthesis and the Decision Heuristic
| Dimension | Systematic | Narrative | Scoping | Meta-analysis |
|---|---|---|---|---|
| Research question | Specific, focused | Broad, contextual | Broad, exploratory | Specific, quantitative |
| Protocol | Pre-specified | Optional | Optional | Pre-specified |
| Search strategy | Exhaustive | Selective | Comprehensive, iterative | Exhaustive |
| Quality appraisal | Mandatory | Informal | Not required | Mandatory |
| Synthesis | Qualitative or quantitative | Qualitative (thematic) | Descriptive mapping | Quantitative (statistical pooling) |
| Reproducibility | High | Low | Moderate | High |
| Output | Evidence-based conclusions | Conceptual synthesis | Conceptual map | Pooled effect size |
| Typical duration | 6–24 months | 2–8 weeks | 3–6 months | 3–12 months |
A simple decision rule:
- Choose a meta-analysis if quantitative pooling is feasible and the studies report compatible metrics.
- Choose a systematic review if a precise question demands exhaustive, reproducible synthesis.
- Choose a scoping review if the goal is to map an emerging or heterogeneous field.
- Choose a narrative review if the goal is conceptual or theoretical framing without formal synthesis.
The fastest diagnostic pair is: Is the question broad or precise? and Are the outcome metrics commensurable? These two answers isolate the correct method almost every time. Feasibility constraints — time, team size, database access — legitimately shape the choice and should be stated in the methods section rather than concealed.
7. Practical Workflow: An SLR in Computer Science
Adapted from Kitchenham's guidelines for evidence-based software engineering.
- Define the research question using PICO or PEO.
- Develop the review protocol — search strings, databases, inclusion and exclusion criteria, quality-assessment instruments.
- Conduct the search across IEEE Xplore, ACM Digital Library, Scopus, ScienceDirect and Google Scholar.
- Screen records in two phases: title and abstract, then full text.
- Extract data using a standardised form capturing study metadata, methodology and findings.
- Assess quality using validated checklists (JBI, CASP).
- Synthesise findings thematically and, where legitimate, quantitatively via meta-analysis.
- Report following PRISMA 2020 guidelines.
8. Case Study: Review Approaches in Cybersecurity
Research question: What is the effectiveness of deep learning models in detecting zero-day network intrusions?
| Approach | Application | Outcome |
|---|---|---|
| Narrative | Synthesise seminal works on deep learning for intrusion detection | Conceptual framing; dominant architectures identified |
| Systematic | Exhaustive search across CS databases using PRISMA | Reproducible evidence on effectiveness across 47 primary studies |
| Scoping | Map all ML applications in cybersecurity | Underexplored areas identified, such as adversarial robustness |
| Meta-analysis | Pool F1-scores and detection rates across studies | Quantitative estimate of average performance improvement |
Each approach yields a different but complementary contribution. A rigorous research programme may use several sequentially: a scoping review to identify gaps, followed by a systematic review and meta-analysis to address a specific gap.
Pooling F1-scores across studies using different datasets, threat models and train/test splits invites exactly the methodological heterogeneity that inflates I². Report it, stratify by dataset, and resist a single triumphant pooled number.
9. Common Pitfalls
- Conflating a review with an annotated bibliography. A review synthesises; a bibliography lists.
- Absence of a clear research question. Without focus the review is unfocused and impossible to replicate.
- Search bias. Relying on a single database or using overly restrictive terms.
- Failure to appraise quality. Including flawed studies without acknowledging their limitations.
- Selective citation. Citing only studies that support the researcher's hypothesis.
- Misclassification of the review type. Applying systematic methods but reporting them as narrative, or the reverse.
- Ignoring the temporal dimension. Failing to specify and justify the publication window. A review of deep learning that silently begins in 2012 has made a substantive methodological commitment.
10. Summary
- Systematic reviews offer reproducible, protocol-driven synthesis with strong evidentiary claims; the gold standard when the question is specific and bounded.
- Narrative reviews allow flexibility, theoretical depth and contextualisation, suited to emerging or interdisciplinary topics.
- Scoping reviews map heterogeneous literatures to identify conceptual boundaries, definitions and gaps.
- Meta-analyses statistically combine quantitative results to produce pooled estimates and to assess heterogeneity and bias.
- The choice should be guided by the research question, the available evidence and the intended inference.
11. Exam Preparation
- Distinguish the four review types using the comparative table. Examiners commonly test the ability to differentiate methodological characteristics.
- Memorise the PRISMA stages and the Arksey and O'Malley scoping framework.
- Understand fixed-effect and random-effects models and the I² statistic conceptually.
- Apply the decision heuristic to example research questions.
- Know the reporting standards (PRISMA, PRISMA-ScR) and their purposes.
- Review at least one published SLR and one meta-analysis in computer science.
- Practise critiquing reviews for rigour — selection bias, missing quality appraisal, inappropriate synthesis.
11.1 Short-Answer Questions
- Differentiate between a systematic review and a narrative review. Under what circumstances is each most appropriate? 3 marks
- List the five stages of the Arksey and O'Malley framework. Why is the optional sixth stage valuable? 3 marks
- What is the role of PRISMA in literature reviews? Identify two differences between PRISMA and PRISMA-ScR. 3 marks
- Define the I² statistic. What does a value of 80% suggest about heterogeneity? 2 marks
11.2 Long-Answer Questions
- You are researching federated learning for healthcare applications. Compare the suitability of a scoping review versus a systematic review, with reference to the research question, expected evidence base and review objectives. 10 marks
- Explain the difference between fixed-effect and random-effects meta-analytic models. Under what conditions is each preferred, and how does τ² influence the choice? 10 marks
- Critically evaluate: “A systematic review is always superior to a narrative review.” Use examples from computer science. 10 marks
- Describe the methodological steps of a systematic literature review in software engineering, citing Kitchenham's guidelines. What are the major sources of bias at each step? 10 marks
11.3 Numerical Question
Three studies report the following effect sizes and standard errors:
| Study | Effect size (d) | Standard error (SE) |
|---|---|---|
| A | 0.45 | 0.10 |
| B | 0.62 | 0.15 |
| C | 0.38 | 0.08 |
- Compute the weight of each study under a fixed-effect model (wi = 1/SEi²).
- Calculate the pooled effect size.
- Compute Cochran's Q and the I² value.
- Interpret the heterogeneity.
12. Further Reading
- Kitchenham, B., & Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report.
- Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32.
- Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses. PLOS Medicine, 6(7).
- Tricco, A. C., et al. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR). Annals of Internal Medicine, 169(7), 467–473.
- Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. Wiley.
- Page, M. J., et al. (2021). The PRISMA 2020 statement. BMJ, 372, n71.